[rb] Set Ruby tests to rerun when failing #16858

titusfortner · 2026-01-06T16:02:48Z

User description

🔗 Related Issues

Uses #16840 for ruby

💥 What does this PR do?

Ruby should rerun failures with debug enabled

Here's an example of it working:
https://github.com/SeleniumHQ/selenium/actions/runs/20821698874/job/59812871900?pr=16858

Targets fail
Failing targegts are added to build/failures/_run1.txt (nested in build directory since it is in .gitignore)
Targets are rerun with SE_DEBUG=true (output may exceed what GitHub will display in console)
Rerun failures are stored in build/failures/_run2.txt
The logs for failures are copied into build/failures directory and renamed to the target name
Failed logs are zipped and made available for download along with _run files

PR Type

Enhancement

Description

Enable Ruby test reruns with debug output on failures
Add rerun-with-debug: true to all Ruby CI workflows
Reduce flaky test attempts from 3 to 2 in local/remote tests
Align Ruby CI with existing rerun failure debugging pattern

Diagram Walkthrough

flowchart LR
  A["Ruby CI Workflows"] -->|"Add rerun-with-debug"| B["Unit Tests Job"]
  A -->|"Add rerun-with-debug"| C["Local Tests Job"]
  A -->|"Add rerun-with-debug"| D["Remote Tests Job"]
  C -->|"Reduce attempts 3→2"| E["Flaky Test Handling"]
  D -->|"Reduce attempts 3→2"| E

File Walkthrough

Relevant files

Configuration changes

ci-ruby.yml `Add debug rerun support to Ruby CI workflows` .github/workflows/ci-ruby.yml Added `rerun-with-debug: true` parameter to Unit Tests job configuration Added `rerun-with-debug: true` parameter to Local Tests job configuration Added `rerun-with-debug: true` parameter to Remote Tests job configuration Reduced `--flaky_test_attempts` from 3 to 2 in Local and Remote Tests jobs	+5/-2

qodo-code-review · 2026-01-06T16:03:17Z

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance

🟢

No security concerns identified

No security vulnerabilities detected by AI analysis. Human verification advised for critical code.

Ticket Compliance

🟡

🎫 #1234

🔴	Ensure that clicking a link whose `href` contains JavaScript (e.g., `javascript:...`) triggers the JavaScript when using `click()` (regression from Selenium 2.47.1 to 2.48.x), specifically observed in Firefox 42.
	Provide/maintain a test case demonstrating the expected alert behavior for the provided reproduction.

🟡

🎫 #5678

🔴	Address the "Error: ConnectFailure (Connection refused)" that occurs when instantiating multiple ChromeDriver instances (Ubuntu 16.04, Selenium 3.9.0, Chrome/ChromeDriver 65/2.35).
	Provide guidance or code changes to prevent/recover from the repeated instantiation failure scenario.

Codebase Duplication Compliance

⚪

Codebase context is not defined

Follow the guide to enable codebase context checks.

Custom Compliance

🟢

Generic: Comprehensive Audit Trails

Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.

Status: Passed

Generic: Meaningful Naming and Self-Documenting Code

Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting

Status: Passed

Generic: Robust Error Handling and Edge Case Management

Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation

Status: Passed

Generic: Secure Error Handling

Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.

Status: Passed

Generic: Secure Logging Practices

Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.

Status: Passed

Generic: Security-First Input Validation and Data Handling

Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities

Status: Passed

Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

qodo-code-review · 2026-01-06T16:04:55Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	✅ ~~Restore correct test expectations~~ Suggestion Impact: The commit updated the failing test expectation in element_spec.rb from expecting 'Nope' to expecting 'Hello', matching the suggested fix (and also corrected several other 'nope' expectations in the same file). code diff: key_reporter.send_keys([:shift, 'h'], 'ello') - expect(key_reporter.attribute('value')).to eq('Nope') + expect(key_reporter.attribute('value')).to eq('Hello') end Revert the intentionally failing test expectation in `element_spec.rb` to its correct value to ensure tests pass. rb/spec/integration/selenium/webdriver/element_spec.rb [132] -expect(key_reporter.attribute('value')).to eq('Nope') +expect(key_reporter.attribute('value')).to eq('Hello') `[Suggestion processed]` Suggestion importance[1-10]: 9 __ Why: This is a critical suggestion as merging the intentionally failing test would break the build on the main branch.	High
General	Define rerun-with-debug input Ensure the reusable `bazel.yml` workflow is updated to accept and process the `rerun-with-debug` input parameter. .github/workflows/ci-ruby.yml [76] -rerun-with-debug: true +# In .github/workflows/bazel.yml: +on: + workflow_call: + inputs: + rerun-with-debug: + description: "Enable rerun with debug logs" + required: false + type: boolean +# Then use it in the test step: +- name: Run Bazel tests + run: \| + bazel test \ + --keep_going \ + --rerun_with_debug=${{ inputs.rerun-with-debug }} \ + ... + `[To ensure code accuracy, apply this suggestion manually]` Suggestion importance[1-10]: 7 __ Why: This suggestion correctly points out that for the `rerun-with-debug` parameter to work, the called reusable workflow (`bazel.yml`) must be updated to accept it, which is not part of this PR.	Medium
Update

titusfortner · 2026-01-12T19:20:08Z

This is failing for #16896

qodo-code-review · 2026-01-13T19:05:28Z

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance

🟢

No security concerns identified

No security vulnerabilities detected by AI analysis. Human verification advised for critical code.

Ticket Compliance

🟡

🎫 #1234

🔴	Fix regression where `click()` on a link with JavaScript in `href` should trigger the expected JavaScript (alerts work in 2.47.1 but not in 2.48.x) in Firefox 42 on the described environment.
	Validate the provided reproduction/test case behavior (from the linked files) works correctly after the fix.

🟡

🎫 #5678

🔴	Investigate and fix repeated ChromeDriver instantiation leading to "Error: ConnectFailure (Connection refused)" on Ubuntu 16.04 with Selenium 3.9.0 / Chrome 65 / ChromeDriver 2.35.
	Provide resolution steps or code changes that prevent the ConnectFailure errors on subsequent driver instantiations.

Codebase Duplication Compliance

⚪

Codebase context is not defined

Follow the guide to enable codebase context checks.

Custom Compliance

🟢

Generic: Comprehensive Audit Trails

Objective: To create a detailed and reliable record of critical system actions for security analysis
and compliance.

Status: Passed

Generic: Meaningful Naming and Self-Documenting Code

Objective: Ensure all identifiers clearly express their purpose and intent, making code
self-documenting

Status: Passed

Generic: Robust Error Handling and Edge Case Management

Objective: Ensure comprehensive error handling that provides meaningful context and graceful
degradation

Status: Passed

Generic: Security-First Input Validation and Data Handling

Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent
vulnerabilities

Status: Passed

⚪

Generic: Secure Error Handling

Objective: To prevent the leakage of sensitive system information through error messages while
providing sufficient detail for internal debugging.

Status:
Debug output exposure: Enabling rerun-with-debug: true may increase the chance that stack traces or internal
details are emitted into CI logs/artifacts, which requires verification that such output
is not user-facing and is appropriately controlled.

Referred Code

rerun-with-debug: true
run: >

Generic: Secure Logging Practices

Objective: To ensure logs are useful for debugging and auditing without exposing sensitive
information like PII, PHI, or cardholder data.

Status:
Potential sensitive logs: The new rerun-with-debug behavior can produce more verbose logs and zipped failure
artifacts, so it should be verified that no secrets/PII can be captured or uploaded from
test runs.

Referred Code

rerun-with-debug: true
run: >
  bazel test
  --keep_going
  --build_tests_only
  --flaky_test_attempts 2
  --local_test_jobs 1

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

qodo-code-review · 2026-01-13T19:06:23Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Enable flaky test retries for unit tests Add `--flaky_test_attempts 2` to the `bazel test` command for `Unit Tests` to align with other test jobs and improve efficiency by enabling individual test retries. .github/workflows/ci-ruby.yml [79-82] ... ruby-version: ${{ matrix.ruby-version }} + rerun-with-debug: true run: > bazel test --keep_going --build_tests_only ++ --flaky_test_attempts 2 --test_size_filters small,medium //rb/... ... `[To ensure code accuracy, apply this suggestion manually]` Suggestion importance[1-10]: 7 __ Why: This suggestion correctly identifies that the `Unit Tests` job is missing the `--flaky_test_attempts` flag, which is inconsistent with the changes made to other test jobs in this PR. Applying this change would improve efficiency by retrying individual flaky tests before rerunning the entire job.	Medium
More

selenium-ci added C-rb Ruby Bindings B-build Includes scripting, bazel and CI integrations labels Jan 6, 2026

qodo-code-review bot added the Review effort 3/5 label Jan 6, 2026

titusfortner changed the title ~~[build] Set Ruby tests to rerun when failing~~ [rb] Set Ruby tests to rerun when failing Jan 6, 2026

titusfortner force-pushed the rb_rerun branch 3 times, most recently from 36c96ca to a70bd8b Compare January 9, 2026 16:58

titusfortner marked this pull request as draft January 10, 2026 01:23

titusfortner mentioned this pull request Jan 10, 2026

[rb][py][dotnet][java][js] move logic for reruns and collecting logs into separate scripts #16879

Merged

titusfortner force-pushed the rb_rerun branch from 8b7b199 to 79a2ff0 Compare January 11, 2026 16:00

titusfortner added 3 commits January 13, 2026 09:06

[rb] rerun failing tests with debug

3239c15

break things

333eb6c

fix things

704e1a2

titusfortner force-pushed the rb_rerun branch from 22b5230 to 704e1a2 Compare January 13, 2026 15:06

Merge branch 'trunk' into rb_rerun

92f5fa9

titusfortner marked this pull request as ready for review January 13, 2026 19:05

qodo-code-review bot added the Review effort 2/5 label Jan 13, 2026

titusfortner merged commit 29eb74b into trunk Jan 13, 2026
35 checks passed

titusfortner deleted the rb_rerun branch January 13, 2026 19:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[rb] Set Ruby tests to rerun when failing #16858

[rb] Set Ruby tests to rerun when failing #16858

titusfortner commented Jan 6, 2026 •

edited by qodo-code-review bot

Loading

Uh oh!

qodo-code-review bot commented Jan 6, 2026 •

edited

Loading

Uh oh!

qodo-code-review bot commented Jan 6, 2026 •

edited

Loading

Uh oh!

titusfortner commented Jan 12, 2026

Uh oh!

qodo-code-review bot commented Jan 13, 2026

Uh oh!

Uh oh!

qodo-code-review bot commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

[rb] Set Ruby tests to rerun when failing #16858

[rb] Set Ruby tests to rerun when failing #16858

Conversation

titusfortner commented Jan 6, 2026 • edited by qodo-code-review bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

🔗 Related Issues

💥 What does this PR do?

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

qodo-code-review bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Compliance Guide 🔍

Uh oh!

qodo-code-review bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Code Suggestions ✨

Uh oh!

titusfortner commented Jan 12, 2026

Uh oh!

qodo-code-review bot commented Jan 13, 2026

PR Compliance Guide 🔍

Uh oh!

Uh oh!

qodo-code-review bot commented Jan 13, 2026

PR Code Suggestions ✨

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

titusfortner commented Jan 6, 2026 •

edited by qodo-code-review bot

Loading

qodo-code-review bot commented Jan 6, 2026 •

edited

Loading

qodo-code-review bot commented Jan 6, 2026 •

edited

Loading